PPC Calibration plots #352
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #352 +/- ##
==========================================
- Coverage 98.60% 96.28% -2.33%
==========================================
Files 35 36 +1
Lines 5533 5673 +140
==========================================
+ Hits 5456 5462 +6
- Misses 77 211 +134 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
ExamplesThese should allow for some tests of these functions. Creating example datalibrary(bayesplot)
ymin <- range(example_y_data(), example_yrep_draws())[1]
ymax <- range(example_y_data(), example_yrep_draws())[2]
# Observations and posterior predictive probabilitites.
y <- rbinom(length(example_y_data()), 1, (example_y_data() - ymin) / (ymax - ymin))
prep <- (example_yrep_draws() - ymin) / (ymax - ymin)
groups <- example_group_data()PAVA Calibration overlayBasic ppc_calibration_overlay(y, prep[1:50,])
Grouped ppc_calibration_overlay_grouped(y, prep[1:50,], groups)
PAVA CalibrationThis isn't yet quite what we want. Now the interval is not what we show in the paper. There, we use consistency intervals, that is, intervals centered at the diagonal displaying, where the calibration curve should lie, i.e. the posterior mean should stay within these bounds. ppc_calibration(y, prep)
ppc_calibration_grouped(y, prep, groups)
|
There was a problem hiding this comment.
This all sounds good, thanks @TeemuSailynoja. I made a few small review comments/questions. In addition to those questions, when you say
This isn't yet quite what we want. Now the interval is not what we show in the paper.
you mean that we will want to change this to use the consistency intervals you use in the paper, right? Do you think it's at all useful to give the user the option to choose which kind of interval? Or just strictly better to use the consistency intervals? I hadn't really thought about that.
| if (requireNamespace("monotone", quietly = TRUE)) { | ||
| monotone <- monotone::monotone | ||
| } else { | ||
| monotone <- function(y) { | ||
| stats::isoreg(y)$yf | ||
| } | ||
| } |
There was a problem hiding this comment.
Is there an advantage to using monotone::monotone instead of stats::isoreg?
There was a problem hiding this comment.
That is, does it do something slightly better? Or the same thing more efficiently? I've seen stats::isoreg before but I had never seen the monotone package. If there's no difference then it's probably not worth checking for the monotone package. If it's better then we could put monotone in Suggests and then check for it like you do here.
| #' @rdname PPC-calibration | ||
| #' @export | ||
| ppc_calibration_overlay <- function( | ||
| y, prep, ..., linewidth = 0.25, alpha = 0.5) { |
There was a problem hiding this comment.
So for these functions prep is a matrix of probabilities and not actually a matrix of draws of binary outcomes from the posterior predictive distribution, right? I think in that case the argument name prep makes sense. But the description at the top of the file says
Assess the calibration of the predictive distributions
yrepin relation to the data `y'
which makes it sound like the user should give us yrep. So I think we just need to reconcile how we describe this to the user.
This is my work in progress of the pava calibration plots discussed in #343
Currently implemented:
ppc_calibration_overlay()ppc_calibration_overlay_grouped()ppc_calibration()ppc_calibration_grouped().ppc_calibration_data()- internal functionNeeds:
ppc_calibration().ppc_calibration_data()be exposed to users?